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SYSTEMATIC VARIATIONS OF INSTRUCTIONAL VARIABLES 
ON LEARNER PERFORMANCE: AIRCRAFT INSTRUMENT COMPREHENSION TASK 



Although many instructional variables have been shown to facilitate 
learning, few of these variables appear to make a consistent contribution 
to learner performance under a variety of instructional conditions. And, 
the lack of comparability across studies that have attempted to examine the 
effects of these variables makes it difficult to determine the contribution 
of each of these variables to the effectiveness of instructional materials 
and procedures. The present research bridges this gap in conventional 
instructional research by systematically manipulating potentially powerful 
variables under a common set of experimental procedures. 

The primary focus of the present research is to determine ^^yhat works 
in effective instruction." The fundamental emphasis of the present ap- 
proach is upon using effective instruction as the research vehicle for 
determining the variables of which instructional effectiveness is a func- 
tion. The overriding emphasis of previous instructional research efforts 
has been upon the variables per se , with little regard for carefully de- 
fining the experimental task and then providing methods and materials that 
ensure learner proficiency on the task. In contrast, the present research 
strategy employs a systematically developed instructional program of 
empirically established effectiveness. Two tryouts of the aircraft instru- 
ment comprehension program were conducted to verify its effectiveness in 
consistently producing criterion-level proficiency in target population 
learners. The program is designed to train learners to identify which one 
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of four pictures of aircraft in flight most nearly corresponds to the posi- 
tion indicated on a panel of attitude and heading instruments. For each of 
the three positional concepts taught in the program (roll, pitch, and 
heading), instruction consists of providing the learner with the rule for 
idf tifying examples of the concept, along with approximately five examples 
of the concept. Practice exercises require the learner to identify the 
picture of an aircraft that most closely approximates the position of the 
aircraft represented by the attitude and heading instruments. Learners rey 
spending to each practice exercise receive immediate feedback for their 
responses in the form of knowledge of correct response. 

Using the aircraft instrument comprehension task, the individual and 
combined effects of practice, incentive, instruction, and feedback on 
learner posttest performance were investigated in a series of four facto- 
rial studies. Two distinct learner populations, AFROTC cadets and graduate 
education students, were studied in isolating the effects of these vari- 
ables. Learner posttest performance was reflected by two measures: 
accuracy, or the number of correct responses emitted, and time required to 
respond to all items on the posttest. 

It was expected that the output of this line of research would result 
in an increased knowledge of the individual and combined effects of vari- 
ables that are under the control of the instructional developer. This 
knowledge should facilitate future attempts at designing instructional pro- 
ducts that are both effective and efficient in producing desired learning 
outcomes . 
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EFFECTS OF PRACTICE AND INCENTIVE 
EXPERIMENT I 

Experiment I was conducted to determine the individual «nd combined . 
effects of practice and incentive in facilitating learner posttest perform- 
ance on the aircraft instrument comprehension task. 

Method 

Subjecte— Forty-eight male Air Force Reserve Officer Cadets enrolled 
in their third year of studies at Arizona State U-^iversity served as the 
subjeci-s in this stu(<y. All cadets in the study had previously taken the 
Air Force Officer Qualifying Test. 

Materials— Variations of the self-instructional materials were studied. 

The program was designed to achive the following instructional objective: 

Given four illustrations of aircraft in roll, pitch, and 
heading, the student will identify the illustration that most 
nearly represents the position indicated on a compass and an 
artificial horizon. An acceptable performance will consist 
of identifying at least 90% of the correct response alterna- 
tives on a 36-item instrument comprehension test. 

Instructions in the program consisted of one instructional cue and 
three examples for each of the three concepts presented: pitch, bank, and 
heading. Eight examples were also presented in which these concepts were 
combined. Practice consisted of one to four practice items for each concept, 
followed by additional 10 practice items at the end of the program. All 
practice items required Ss to identify which of two or more drawings of an ^ 
aircraft in flight most nearly represented the position shown on an attitude 
indicator and a heading indicator. Sample instructional materials are con- 
tained in Appendix A; sample practice items may be found in Appendix B. 
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Upo^i. completion of each practice iten, feedback was available to S by 
nioving.a !paper S;lide 

. Procedure— Ss were assigned to. treatment groups by a pirocedure that 
inade .it unlikely that Ss in the no incentive conditions were informed of 
the incentive being used in pther groups, Since Ss attended classes in 
four different sections that met on two consecutive days, all Ss in the 
-first two sections were assigned to the no incentive conditions,* and all Ss 
iii the subsequent two sections were assigned to the incentive conditions. 
Within; each class section, half of the |s Were randomly assigned to the 
practice conditions, and the remaining half were assigned randomly to the 

novpraictice conditions. 

All Ss were administered a pretest one week prior to initiating the 
experiment. Ss were allowed a maximum of three minutes to complete the 
nine pretest items. During each experimental session, S read the instruc- 
' tional materials and responded to items on a posttest. A proctor displayed 
the time on 8Js" x 11" cards at the front of the room at 0.25 minute inter- 
vals throughout the experimental session. Ss were instructed to complete 
the instructional booklet assigned to them, and to record their completion 
times for the instructional booklets. Proctors collected the booklets as 
the Ss finished them. The posttest was administered after all Ss had com- 
pleted the instructional booklets. Ss were again asked to record their com- 
pletion times, and the test booklets were collected as each S finished. 

Ss in the incentive conditions received written instructions located 
at the beginning of their instructional booklets indicating that they could 
earn up to nine quiz points on their course grade by responding accurately 
to the items on the posttest. In addition, the instructions indicated that 



the four cadets who responded to the ppsttest items with the greatest 
accuracy, and in the least amount of time, would have an opportunity to 
"fly" a formation trainer at Williams Air Force Base, Arizona. Instruct 
tions to Ss in the no incentive conditions simply stated that their grades' 
in the course would not be affected by their participation in fe=e."tryout" 
and that the developers of the self- instructional program would "appreciate 
their best efforts." 

Ss in the no practice conditions received instructional booklets in 
which all practice items were deleted from the program. Booklets for each 
S in this condition contained only the instructional cues and examples for 
each concept taught in the program." The booklets for Ss in the practice 
conditions contained approximately twice as many pages as the booklets that 
did not include practice items. 

Criterion Measures— The posttest contained directions, a sample test 
item, and 36 multiple-choice items. Appendix C contains a sample test item. 
All test items required S to identify which one of fo.ur drawings of an air- 
craft in flight most nearly represented the position shown on an attitude 
indicator and a heading indicator. A pretest consisting of nine items 
selected to be representative of items on the posttest was also employed. 

The test item pool consisted of 72 different positions of an aircraft 
in flight. Three dimensions of the aircraft's position (pitch, roll, and 
heading) were systematically vaned to create the item pool. Three levels 
of pitch were used: level, climb, and dive. Three levels of roll were 
used: no bank, 30° right bank, and 30° left bank. Eight levels of heading 
were used: four primary compass headings (N, S, E, W) and four intermedi- 
ate compass headings (NE, SE, NW, SW). As described in the validation 
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report (Higgins, Kearns. and Tenpas. 1974). the posttest items were 
systanatically selected to represent equally the variations in heading, 
pitch and roll . 

In addition to recording the number of correct responses emitted by 
each S on the posttest, the amount of time that S required to complete the 
items on the posttest was recorded to the nearest 0.25 minute. This was 
accomplished by noting the simultaneous starting time of all Ss and re- 
quiring each S to record on his answer sheet the time shown on a card held 
by a proctor at the front of the room. Anecdotal observations by two other 
proctors in the room indicated that S's self-recordings were veridical. 

Data Analyses- The pretest-treatment-posttest experimental arrangement 
constituted a 2 x 2 factorial design with 12 Ss in each of the four cells. 
Analyses of variance were made to determine the individual and combined 
effects of practice and incentive on posttest scores and posttest times. 
Each analysis was evaluated at the .05 level of confidence. 

Results 

An analysis of scores on the 9-item pretest revealed that the mean 
scores across the treatment groups varied less than 0.25 points from the 
overall mean pretest score of 6.25 (Table 1). As can be seen from Table 2, 
these differences were not statistically significant. These results add 
confidence to our assumption of equivalence across treatment groups with 
respect to S's entry skills on the experimental task. 

The mean posttest scores of the treatment groups (Table 3) were all 
within one-point of the overall mean score of 33.31. F-ratios for incen- 
tive, practice, and the practice X incentive interaction were not 
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TABLE 1 



Pretest Scores {\mber corrfect of 9 ^possible points) by Treatipent 



Incentive 



Practice 



Practice 



No Practice 



Totals 



Incentive 


6.42 


6.08 


6.25 


U-f:-. No Incentive 


6.25- 


6.25 


6.25 


totals 


6.33 


6.17 


6.25 



TABLE 2 

Analysis of Variance: Pretest Scores 



Source of Variation 




df 


MS 


F-Ratio 




- Incentive 


o.oo 


1 


0.00 


0.00 


NS* 


Practice 


.33 


1 


.33 


.07 


NS 


Incentive X Practice 


.33 


1 


.33 


.07 


NS 


I Within 


200.33 


44 


4.55 







*NS = Not Significant 
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TABLE 3 

Posttest Mean Scores (nuiiter correct of 36 possible points) by Treatment 



Practice 



Incentive 


.Practice 


No Practice 


Totals 


Incentive 


33.33 


32.42 


32.88 


No Incentive 


33.50 


34.00 


33.75 


Totals 


33.42 


33.21 


33.31 



TABLE 4 

Analysis of Variance: Posttest Scores 



Source of Variation 


SS 


df 




F-Ratio 




Incentive 


9.19 


1 


.9.19 


1.12 


NS 


Practice 


.52 


1 


.52 


.06 


NS 


Incentive X Practice 


6.02 


1 


6.02 


.73 


NS 


Within 


360.58 


44 


8.20 




V 
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statistically significant (Table 4). A performance gain was observed be- 
tween the pretest scores and the posttest scores. On the pretest, Ss 
answered a' mean of 69.4% of the items correctly; Ss answered a mean of 
92.5% of the items correctly on the posttest. 

|s spent differential amounts of time answering" items on the posttest. 
Mean posttest times by treatment are reported in Table 5. Posttest time 
differences were analyzed using an analysis of variance, as shown in Table 
6. Statistically significant differences were found for both practice 
:(F = 22.3, df = 1/44, £ < .001) and for incentive ( F = 15.4, df = 1/44, 
£< .001),. The practice X incentive interaction was not significant. 

A further analysis was made to determine whether the experimental 
variations facilitated attainment of the task objective. The objective 
indicated that Ss should be able to respond accurately to at least 90% (or, 
32 out of 36) of the items on the posttest. No appreciable variation in 
the number of S^s who exceeded this performance level was observed that 
might be attributed to either practice or incentive. However, an increase 
in the number of Ss exceeding the 90% performance level on the pretest, 
while 81% exceeded this level on the posttest. Of the 12 Ss in the treat- 
ment group that received neither practice nor incentive, 4 (33%) of the Ss 
exceeded the prespecified 90% performance level on the pretest, but 10 
(83%) of the Ss attained the objective on the posttest. 

Discussion 

Experiment I was conducted to determine the individual and combined 
effects of practice and incentive in facilitating learner posttest perform- 
ance on the aircraft instrument comprehension task. The results indicated 
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TABLE 5 

Mean Posttest Times (minutes) treatment 



Incentive 



Practice 



Practice 



No Practice 



Totals 



Incentiye 
No' Ihcentiye 
totals 



5.98 
7.79 
6.88 



8,13: 

9; 60. 
. 8.86 



7.05 
8; 70 
7v88 



TABLE 6 

Analysis of Variance: Posttest Times 





Source of Variation 


SS 


df 


MS 


£-Ratio 




Incentive 


32.50 


1 


32.50 


15.42* 




Practice 


47.00 


1 


47.00 


22.31* 




Incentive X Practice 


0.09 


1 


0.09 


0.04 


h ' 


Within 


93.02 


■ 44 


2.11 




\ ' \ 


*£ < .001 
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ll!Vt neith^^ significantly affected the accuracy^ of 'pbsttest 
W'^:'"' '^Pl^nce,. is in the treatment group that simply read, the instructional 

fi;.-^ ^ "^^^ score of 34 on the 36-item posttest. Such a high mean 

, .PQsti^est score created a ceiling effect that left little rdbm for improve- 

:t ""^"^ that' could be attributed to either practice or incentive. That is^ 

g;> the increase that was observed bfitween pretest and posttest scores appeared 

JsVv; attributable tb the instructional materials per se ; the experimental 

IV variables had little effect in controlling the accuracy of posttest perform- 

|::,. ance. Also, a pretest-posttest comparison of the number of subjects 

i:.; attaining, the objective showed an increase that appeared to be attributable 

i : to the instructional materials. 

p.% An analysis of the time required for learners to complete the posttest 

' ■ showed a different picture. The amount of time required for learners to 

J:- * complete the posttest was significantly reduced both by the incentive that 

L ^'as offered, and by the practice provided in the instructional program. In 
addition to being a criterion measure that appears less susceptible to the 

\ ■ ceiling effects imposed by posttest scores, posttest time may have consider- 

, able external validity with respect to pilot training. Indeed, the speed 

I with which an individual can accurately read and respond to instruments may 

I prove an important predictor of later pilot effectiveness. 



EFFECTS OF INSTRUCTION AND FEEDBACK CONTINGENCY- 
EXPERIMENT II ■ 

Previous studies indicate that feedback in the form of knowledge of 
the correct resppnse^^^^^^^ enhance learner performance on the air- 

craft Inst wnt. c^^^^ beyond that attributable to; instruction 

alone. -Also,. 4he results of experiment I suggested that instruction per se 
is the iiwapr factor controlling learner posttest accuracy scores. HpweVer, 
these studjw^^^^ slidie device to present KCR, and fany of the 

learner^. Were observed to ignore the slide and proceed through the program 
Wittput receiyihg/feedback, It is indeed possible that KCR failed to 
significantly enhance, learner performance simply because the learners found 
it "inconvenient" to move the paper slide in order to obtain this feedback. 

the purpose of Experiment II was to determine the individual and com- 
bined-effects of instruction and the contingencies for obtaining feedback 
on learner posttest performance. Also, in order to reduce the ceiling 
effect that was observed with posttest scores in Experiment I, an effort 
was made to procure learners that were expected to be less skilled at the 
aircraft instrument comprehension task than AFROTC cadets. Two levels of 
instruction were examined: instruction and the absence of instruction. 
The two types of feedback contingencies employed in the study were KCR re- 
• ceived immediately contingent on responding to practice exercises, and KCR 
received contingent on moving a paper slide device to reveal the correct 
response after responding. 



Method 

Subjects- Thirty-Six Arizona State University graduate education stu- 
dents enrolled during the spring semester of 1974 served as the subjects in 
this study. 

Materials— The aircraft instrument comprehension program .described in 
Experiment I was used. Half of the Ss marked their program responses on 
chemically treated answer sheets (A. B. Dick Company) designed to provide 
KCR. A latent image in the form of an "X" appeared inmediately when ^ 
touched the appropriate space with the special pen provided. Program re- 
sponse sheets and marking pens app.eared identical for all Ss, regardless of 
the experimental condition, is who did not receive the chemically treated 
answer sheets were required to manipulate a paper slide device in order to 
obtain KCR. 

Procedure— Ss were informed that their participation in the "tryout" 
was appreciated and that the developers of the self-instructional- program 
would "appreciate their best efforts." Other than an attempt to appeal to 
the education majors' interest in the quality of instructional products, 
extrinsic incentives were offered contingent on learner performance. Also, 
Ss were assumed to be naive with respect to the aircraft instrument compre- 
hension task; consequently, no pretest was administered. 

Upon entering their classroom on the day of the study, Ss were randomly 
assigned to one of four treatment groups. Each treatment group received a 
different set of materials. Ss in the instruction groups received the com- 
plete set of instructional booklets described in Experiment I. Ss in the 
no instruction groups received an instructional booklet with all 
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instructional cues and examples removed. All Ss began work on the program 
at the same time and were instructed to record the time on their answer 
sheets and raise their hands when they were finished. A proctor displayed 
the time at 0.25 minute intervals throughout the program and posttest. 
Following completion of the program, each S's instructional booklet was re- 

* 

moved and replaced with a test booklet, answer sheet, and marking pen. The 
time each S began the posttest was recorded on his/her answer sheet by a 
proctor. Directions on the first page of the posttest booklet indicated 
that Ss were to work as quickly as possible without sacrificing accuracy. 
■Ss recorded the time they finished the test on their answer sheets; they 
were free to leave as soon as proctors collected their test materials. 

Criterion Measures- Posttest time and accuracy were recorded for each 
Sj, as described in Experiment 1. 

Data Analysis- The posttest-only control group arrangement resulted in 
a 2 X 2 factorial design, with 9 Ss in each of the 4 cells. Analyses of 
variance were used to determine the individual and combined effects of in- 
struction and feedback contingency on posttest scores and posttest times. 
Each of the analyses was evaluated at the .05 level of confidence. 

Results 

Tables 7 and 8 show a significant main effect for instruction on the 

posttest scores (F = 8.91, df = 1/32, £ < .01). Ss in the groups receiving 

instruction, regardless of the feedback contingency employed, scored an 

average of 5.66 points higher on the 36-item posttest than S in the nojn- 

struction conditions. Differences in posttest time were not statistically 

significant, as shown in Tables 9 and 10. An inspection of individual S 
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TABLE 7 

Posttest Mean Scores (number correct of 36 possible points) 







Feedback Continaencv 




Instruction 


Kesponse 
Contingent 


:)noe-use 
Continqent 


Totals 


Instruction 


25.41 


21.71 


23.72 


No Instruction 


16.44 


15.64 


16.05 


Totals 


21.05 


18.72 


13.24 



N = 9 per cell 



TABLE 8 

Analysis of Variance; Posttest Scores 



Source of Variation 


MS 


DF 


F-Ratio 




Instruction 


592.00 • 


1 


8.91* 




Feedback Contingency 


49.00 


1 


.83 


NS ■ 


Instruction X Feedback 


21.74 


■ 1 


.37 


NS 


Within 


59.37 


32 






Total 


71.43 


35 







*£ < .01 
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TABLE 9 

Mean Posttest Times (minutes) by Treatment 



Fppriha r.k Contingency . . 

Slide-use 
Contingent 



TABLE 10 

Analysis of Variance: Posttest Times 



Source of Variation 



MS. 



DF 



F-Ratio 



Instruction 


.03 


1 


.00 


NS 


Feedback Contingency 


10.56 


1 


.33 


NS 


Instruction X Feedback 


11.67 


1 


.36 


NS 


Within 


32.18 


32 






Total 


30.06 


35 
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performance scores revealed that only 7 of the 3b Ss exceeded the minimum 
performance level of 90% correct on the posttest. All 7 of these Ss were 
in treatment groups that received instruction. 



Discussion 

Experiment II was conducted to investigate the individual and contined 
effects on instruction and feedback contingency on the speed and accuracy 
of learner posttest performance. The results indicated a significant ef- 
fect 'for instnjction; neither the main effect for feedback contingency nor 
the instruction X feedback interaction were statistically significant. 
However, the generality of these results were quite limited by the fact 
that less than 40% of the learners receiving instruction exceeded the mini- 
mal performance score of 32 on the 36-item posttest. These results suggest 
that the aircraft instrument comprehension prograir may not be effective in 
attaining the task objective with relatively naive learners. However, an 
incentive was not provided for the learners to "try their best" on the 
materials; this may have been responsible for the low overall performance 
that was observed. Additional data with similar learners is obviously in- 
dicated in order to clarify these ambiguous results. 
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EFFECTS OF INSTRUCTION AND PRACTICE 
EXPERIMENT III 

the results of Experiment I indicated that practice and incentive do 
riot significantly enhance the accuracy of learner posttest responses on the 
aircraft instrument comprehension task. Based on these results, it was 
suggested that instruction may be a major factor in nvaintaining high post- 
test performance levels. However, the learners in that experiment were 
alreaOy reasonably proficient at performing the task being taught, as indi- 
cated by their high pretest scores. Perhaps practice would significantly 
enhance posttest performances in cases where learners were not familiar 
with the task. The purpose of Experiment III, therefore, was to detemine 
the individual and combined effects of instruction and practice on the 
posttest performances of relatively "naive" learners. 

Method 

Subjects— Fifty- two undergraduate students enrolled in an educational 
psychology course at Arizona State University during the spring semester 
of 1974 served. 

Materials- The materials in this study remained identical to those de- 
scribed in Experiment II. 

Procedures— Upon entering their classroom on the day of the study, Ss 
were randomly assigned co one of four treatment- groups . Each treatment 
group received a different set of materials. The in<;truction and practice 
group received an instructional booklet with all instructional cues, 
examples, and practice items intact. The instruction only group received 
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ah instructional booklet whh all the practice items deleted. The no 
practice>no instruction grou.j recejved only a posttest booklet; Ss did not 
receive instructional cues, exajiiples, or practice items. 

All Ss received oral instructions indicating that they were parti ci- 
■pating in an experinient, and that they would receive, extra credit toward 
their grade in the course in return for their participation* Also, the. 
first page of each instructional booklet indicated that Ss could earn up to 
nine points of extra credit if they responded quickly and accurately on a 
subsequent posttest. The instructions also indicated that various types of 
materials had been distHbuted and that each S would be competing for extra 

» 

credit only agai.jst other Ss that received the same materials. 

the last page of each instructional booklet directed S^ to record the 
time he had finished his booklet (as displayed on the time card held by a 
proctor at the front of the room), and to raise his hand so that a proctor 
could collect his materials. The proctor checked the completion time S had 
listed, collected the materials, and instructed Ss who had taken the post- 
test that they could leave. Ss who had completed a version of the instruc- 
tional program were given a copy of the posttest by the proctor, who 
recorded S.*s starting time. When Ss finished the posttest^ he recorded his 
completion time, had his material collected by a proctor,and was dismissed. 

Criterion Measures— The criterion measures remained identical to those 
described in Experiment I. 

Design and Data Analyses -The posttest-only control group arrangement 
constituted a 2 x 2 factorial design. Analyses of variance were made to 
determine the individual and combined effects of instruction and practice 
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on posttest scores and posttest time. The results of each analysis were 
evaluated at the .05 level of confidence. 

Results 

Table 11 shows the mean posttest scores by treatment. The mean score 
for groups receiving instruction was almost 14 points higher than the mean 
score for groups not receiving instruction. Differoices in mean scores be- 
tween groups receiving practice and groups not receiving practice was less 
than one point. A two-way analysis of variance (Table 12) revealed a 
statistically significant difference attributable to instruction (£= 82,62, 
df = 1/48, £< .001). F-ratios for practice and the practice X Incentive 
interaction were not statistically significant. A further inspection of 
the posttest scores revealed that 11 (42%) of the Ss In the group that re- 
ceived instruction exceeded the 90% minimum performance level established 
■for the task. 

Mean posttest times by treatment are shown in Table 13. Table 14 in- 
dicates that the F-ratios for instruction, practice, and their interaction 
were not statistically significant with respect to posttest time. 

Discussion 

Learners that received instruction scored significantly higher on the 
posttest than learners who did not receive instruction. Neither the pre- 
sence of instruction nor the combination of instruction and practice 
significantly enhanced learner performance beyond the effects attributable 
to instruction alone. 
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TABLE 11„ 

Posttest Mean Scores (number correct of 36 possible points) by Treatment 



Practice 



Instruction Totals 

Practice No Practice 



Instruction 28.07 31.53 29.80 

No Instruction 16.61 15.07 15.84 

Totals 22.34 23.30 22.82 



TABLE 12 

Analysis of Variance: Posttest Scores 



Source of Variation 


SS. 


df 


MS 


F-Ratio 




Instruction 


2534.02 


1 


2534.02 


82.62* 




Practice 


12.02 


1 


12.02 


.39 


NS 


Instruction X Practice 


■ 81 .25 


1 


81.25 


2.65 


NS 


Within 


1472.16 


48 


30.67 







*£ < ,001 




TABLE 13 



Mean Posttest Time (minutes) by Treatment 



Practice " ' [ 





instruction 


Practice 


^- No Practice 


Totals 




Instruction 


13.65 


15.51 


14.58 




No Instruction 


13.30 


15.01 


14.15 




totals 


13.47 


15.26 


1.4.36 



TABLE 14 

Analysis of Variance: Posttest Times 



Source of Variation 


SS 


if 


MS 


F-Ratio 




Instruction 


2.32 


1 


2.32 


0.13. 


NS 


Practice 


41.58 


1 


41.58 


2.50 


NS 


Instruction X Practice 


0.'08 


1 


0.08 


0.01 


NS 


Within 


797.44 


48 


16.61 
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These results hardly seera surprising. Instruction should enhance the 
performance of learners beyond that of learners who do not receive such 
instruction. What is significant about these findings, however, is the 
pattern of results that is beginning to emerge from the three studies that 
have been considered thus far. Instruction, when it is well -designed, 
appears to be a variable of such impact that it leaves little room for im- 
provement in learner performance that might be attributable to other refine 
ments such as practice, incentive, and feedback. 

Feedback is a particularly interesting case in point. Recent years 
have produced a plethora of research in the area of feedback effects on 
learner performance (cf . Anderson, 1967; Annett, 1969; Briggs and Hamilton, 
1964; Gagne and Rowher, 1969). And, the results have been far from con- 
si stent in this area: feedback sometimes enhances performance over 
learners that do not receive feedback during instruction; frequently it 
does not. Relatively few of these studies, however, have employed systema- 
tically developed instructional materials (Higgins, 1972). It may be that 
the effectiveness of feedback in enhancing learner performance varies 
directly with the degree to which materials are systematically designed to 
ensure learner attainment of prespecified instructional objectives. 



EFFECTS OF FEEDBACK AND INSTRUCTION 
EXPERIMENT IV 



Experiment IV was conducted to determine the individual and combined 
effects of instruction and feedback in the aircraft instrument comprehen- 
sion program. Two levels of instruction were investigated (presence and 
absence), along with two levels of feedback (also, presence and absence).. 
If instruction is a variable of such import that it negates the effective- 
ness of feedback, as indicated in the above discussion, then feedback 
should be observed to enhance learner performance only in the absence of 
instruction. 

Method 

Subjects;— Sixty- three junior and senior college students, enrolled in 
the Air Force Research Officer Training program at the University of 
Arizona, served as the subjects in this study. All Ss had previously taken 
the Air Force Officer Qualifying Test. 

Materials— All materials remained identical to those reported in the 
previous experiments. 

ProcedureS" Ss were divided between two experimental sessions. During 
each session Ss were randomly assigned to one of four treatment groups: 
instruction and feedback , instruction and no feedback , feedback and no in- 
struction , or no feedback and no instruction . Performance incentives 
contingent upon both speed and accuracy of S^'s posttest responding were 
identical to those described in Experiment I. Ss were informed of the in- 
centives through oral directions given at the beginning of each 
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experimental session. The incentives were identical for all treatment 
groups . 

At the beginning of the session, Ss were given general oral directions 
about the study, a 9-item pretest, and an IBM response sheet. Following 
completion of the pretest, all Ss were given the instructional program 
booklet and either a chemically treated or nonchemically treated response 
sheet. Each ^ received a special marking pen regardless of the nature of 
his response sheet. The feedback groups were instructed to mark the chemi- 
cally treated response sheet as many times as necessary to reveal the KCR 
for each practice item in the program. A standard time lapse of 0=25 
rainute intervals was displayed for all S^s, As Ss completed the instruc- 
tional program, proctors monitored each S^'s recording of his completion 
time, picked up program materials, and distributed the criterion test. 

Criterion Measures— The criterion measures were identical to those 
described in Experiment I. 

Design and Data Analyses— The pretest-posttest experimental arrange- 
ment constituted a 2 x 2 factorial design. Analyses of variance were made 
to determine the individual and combined effects of instruction and feed- 
back on posttest performance and post lest time. The results of each 
analysis were tested at the .05 level of confidence. Group means on the 
pretest were evaluated using an analysis of variance to determine the 
similarity of treatment groups. Ss who received a score of seven or more 
on the pretest were considered to be proficient on the task prior to the 
experiment and were eliminated from all data analyses reported. 
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Results 

The mean score for each group on the 9-item pretest is shown in Table 
15. No statisitically significant differences were observed in pretest 
scores across the treatment groups (Table 16). 

the mean score for each group on the 36-item criterion test is shown 
in Table 17. The mean score for the instruction group was approximately 4 
points higher than the no instruction group; the feedback group scored 
about 2 points higher than the no feedback group. An analysis of these 
scores revealed a significant difference for instruction (F_= 4.73, df = 
. -1/37, £< .05); differences for either feedback or the instruction X feed- 
back interaction were not significant (Table 18). 

Mean times required for each group to complete the posttest are re- 
ported in Table 19. The mean time for the instruction group, 7.14 min, 
was less than the mean time of 8.21 min for the no instruction group. The 
mean time of 7.93 min for the feedback group exceeded the mean time of 
7.30 min for the no feedback group. A two-way analysis of variance (Table 
?0) again revealed a significant difference for instruction (£ = 4.45, 
df = 1/37, £< .05), but no significant differences for either feedback or 
the instruction X feedback interaction were observed. 

Discussion 

The results of Experiment IV indicate that instruction ehances both 
the accuracy and the rate at which the learners responded to items on the 
posttest. However, the anticipated interaction between feedback and in- 
struction was not obtained. Feedback did not enha-.ice posttest performance 
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fe.' . TABLE 15 

Mean. Pretest Scores {nuirber correct of 9 possible points) by Treatment 



Feedback 



Instruction 


Feedback 


No Feedback 


Totals 


Instruction 


5.88 


5.44 


5.66 


No instruction 


5.53 


5.50 ' 


5jJ2 


Totals 


5.70 


5.47 


5.59 



N = 16 per cell 



TABLE 16 

Analysis of Variance: Pretest Scores 



9^ . 














Source of Variation 


SS 






F-Ratio 






Instruction 


0.32 


1 


0.32 


0.13 


NS 


t '' • 


Feedback 


0.89 


1 


0.89 


0.37 


NS 


It, _ 


Instruction X Feedback 


0.65 


1 


0.65 


0.27 


NS 


'JS 


Within 


143.42 


60 


2.39 
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TABLE 17 

Mean Posttest Scores (nunter correct out of 36 possible points) by Treatii«nt 



Feedback 



> 


Instruction 


Feedback 


No Feedback 


Totals 




Instruction 


32.32 


31 .73 


32.06 


V 


No Instruction 


31.67 


25.24 


27.84 


■> 


Totals 


31.99 


28.48 


29.95 



TABLE 18 

Analysis of Variance: Posttest Scores* 

















Source of Variation 


SS 


df 


MS 


F- Ratio 




1 


Instruction 


18.05 


1 


18.05 


4.73* 






Feedback 


8.69 


1 


8.69 


2.28 


NS 




Instruction X Feedback 


5.07 


1 


5.07 


1.33 


NS 




Within 


140.99 


37 


3.81 







*p < .05 
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TABLE 19 

Mean Posttest Times (Minutes) by Treatment 







Feedback 




Instruction 


Feedback 




Totals 




No Feedback 



Instruction 7.18 7.11 7.14 

Kp Instruction 8.68 7.50 8.21 

Totals 7.93 7.30 7.61 



TABLE 20 

Analysis of Variance: Posttest Tiines 



Source of Variation 


SS 


df 




F- Ratio 




Instruction 


.89 


1 


.89 


4.45* 


NS 


Feedback 


.39 


1 


.39 


1.95 


NS 


Instruction X Feedback 


.30 


1 


.30 


1.50 


NS 


Within 


7.75 


37 


.20 







*p < .05 
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even when instruction was withheld from the learners. These results serve 
to further reinforce the idea that instruction is the predominant variable 
controlling learner performance in the aircraft instrument comprehension 
task. 
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GENERAL DISCUSSION 

The results of Experiments I-IV are sunroarized in Table 21. The most 
salient conclusion that emerges from these data concerns the overpowering 
effect that instruction alone has on learner performance of the aircraft 
instrument comprehension task. In Experiment I, the treatment group that 
•simply read the instructional materials had a mean score of 34 on the 35- 
item posttest. Such a high mean score left little room for improvement 
that could be attributed to either the practice or the incentive variable. 

In Experiment II and III, no significant differences were found for 
either feedback or practice; instruction alone was responsible for the 
observed differences in mean treatment group scores • And, this pattern 
continued in Experiment IV, where feedback failed to contribute signifi- 
cantly to posttest performance— even in the treatment group for which in- 
struction was withheld. 

The present research points to instruction as the variable of singular 
importance in designing effective materials and procedures to facilitate 
desired learning outcomes. This finding, though hardly inconsistent with 
common sense, p'rovides a firm empirical foundation for an assumption that 
permeates all systematic instructional development efforts— instruction 
does indeed make a difference. However, instruction per se is not a unitary 
variable. Rather, instruction is a constellation of variables such cues, 
rules, examples, prompts, fading, successive approximations, information 
redundancy, etc. Clearly, the path is marked for isolating the critical 
variables that constitute effective instruction. The research strategy 
outlined in the present experiments of beginning with an effective 
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||:.{~ , jnstructiohal product should provide a i3rototype for siich future eri- 

I^^V, Although the present research points the way to a detailed functional 

analysis-of instructional variables, it is not clear at present whether the 

i |; results of these studies can be generalized to tasks of greater conplexity 

fj^ than the aircraft instrument comprehension task. Perhaps instruction is 
not a fundamental determiner of learner performance on tasks which require 

f : a shaping of more complex response repertoires, as in the skills we, com- 

monly relegate to the psychomotor domain. Clearly, more research is needed 
in this area to determine the boundary conditions of the effects instruc- 
> tion has in controlling learner behavior. 

1^ ; The following supporting documents are in the R & D Case File at 

Flying Training Division, Air Force Human Resources Laboratory, Williams 

|: Air Force Base, Arizona 85224; 

K Higgins, N. C* Aircraft Instrument comprehension program: FonirB . (AFSC 

: Contract No. F41609-71-C-0027, Task Order No- 3) Tempe. Arizona: 

- Arizona State University, 1973. 

' Higgins, C, and Kearns, D. R. Validation report: Aircraft instrument 

3 comprehension: A self -instructional progranK" Form A . (AFSC Contract 

k \ No. F41609-71-C-0027, Task Order No. 3) Tempe, Arizona : Ari zona State 

5 \ University, 1973. 

Higgins, N. C, Kearns, D. R., and Tenpas, B. G. Validation report: Air- 
h craft instrument comprehension : A self-instructional program: Form 

J >: B. (AFSC Contract No. F41609-71-C-0027,.Task Order No. 3) Tempe, 

Arizona: Arizona State University, 1974* 

? Kearns^ D. R., Tenpas, b. G., and Higgins, N. C* Aircraft instrument com- 

1 prehension test: Form B. (AFSC Contract No. F41609-71-C-0027, Task 

1 - Order No. 3) Tempe, Arizona: Arizona State University, 1973. 



REFERENCES 

Anderson, R. D. Educational psychology. Annual Review of Psychology , 1967, 
18, 129-163. 

Annett,. J. Feedback and human behavior . Baltimore: Penguin Books, 1969. 

Briggs,. L. J., and Hamilton, N. R. Meaningful leami'^g and retention: 

Practice and feedback variables. Review of Educational Research , 1964, 
34, 545-558. 

Gaghe, R. M., and Rowher, W. Instructional psycholoqy. A nnual Review of 
Psychology, 1969, 20, 381-418. 

Higgins, Ni C. Feedback in group Instruction . (AFSC Technical Report No. 
2021) Tempe, Arizona: Arizona State University Instructional Resources 
Laboratory, 1972. 



39 

^ER|c 



APPEI\DIX A 



Sample Instructional Materials 



V' 
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Attitude Indicator-Pitch 



The instrument labeled attitude indicator shows whether the 
airplane is climbing or diving. This instrur.ient also shov/s 
the degree of bank to the right or left. 

The small aircraft silhouette in this instrument remains 
stationary. The position of the heavy black line, representing 
the horizon, varies with the airplane's position. 



lA 




When the airplane is 
flying level, the 
horizon line will be 
directly on the air- 
craft silhouette as 
shown above* 



B 




If the airplane is 
climbing, the silhouette 
is seen between the 
horizon line and the 
triangular pointer. 




attitud;£ \ 

JNDICATC:? i 

! ! 

If the airplane is 
diving, the horizon 
line will be betv/een 
the silhouette and 
the pointer. 



.The airplane in Figure A is 
^'climbing on a northwest heading. 



The horizon line is below 
the aircraft silhouette in 
the attitude indicator. The 
arrow in the heading indicator 
is pointing midway between the 
N and the W. 





ATTJTUDE 




INDICATOR 



The airplane in Figure B is 
diving on a southeast heading. 



The horizon line is above the 
aircraft silhouette in the 
attitude indicator. The 
arrow in the heading in- 
dicator is pointing mid- 
way betv/een the S and the 
E. 
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ATTITUDE 
IWOJCATOR 




HEADING 
INDICATOR 

llB ^ 



i^ieeurtd- fjplow. is moh noarly in. -tl 
■■ M^^.^puc -cho 1 cos r7 • " — ^ 



il^'rf'^''- n*wa!j3^;o'n, the response 
"sKeet. ■ 




2. 




ATTITUDE 
INDJCATOf? 




KEADIWG 
IKDJCATGR 



A 




B 




V - 









3. 
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ATTITUDE 
INDICATOR 



HEADING 
INDICATOR 



mm 



Wbich of the aircrafc pictured bblow is in the position 
shovvn on the instrument panel? 

Mark your choice opposite item number 4 on your 
response sheet • 



A 

n 


B 






c 

n 


D 






APPENDIX C 



Sample Post test Item 
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